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(54) Title: MORPHOLOGICAL CLASSIFICATION SYSTEM AND METHOD 
(57) Abstract 

A method of classifying objects in a specimen includes the steps of obtaining a first digital representation of at least part of 
the cytological specimen, stonng the first digital representation, perfonning a first filtering operation to filter out image3 in the 
tirst representation that are the approximate size of a malignant or premalignant cell or smaller to produce a second digital re- 
presentation, removing the images in the second representation from the images in the first representation to produce a third re- 
presentation, performing a second filtering operation to filter out images in the first representation that are smaller than the ap- 
proximate size of a premalignant or malignant ceil to produce a fourth representation, and eliminating the images in the fourth 
representation from the images m the third representation to produce a representation having substantially only images the ao- 
proximate size of a premalignant or malignant cell. j j ^ tt c p 
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MORPHOLOGICAL CLASSBFICATION SYSTEM AND METHOD 
TECHNICAL F TFi n OF TRE INVENTION / 

This invention relates generally to classification, particularly to cytology, and, 
more particularly, to a method and apparatus for quickly and accurately classifying 
cells based on cellular morphology. 

BACKGROUND OF THE INVENTION 

In the medical industry tiiere is often the need for an experienced laboratory 
technician to review a specimen of biological matter for the presence of cells of a 
certain cellular type. An example of this is the need to review a pap smear slide for 
the presence of malignant or premalignant cells. A pap smear often contains as many 
as 100,000 to 200,000 or more cells and other objects, each of which a technician 
must individually inspect in order to determine the possible presence of very few 
malignant or premalignant cells. Pap smear tests, as well as other tests requiring 
equally exhausting cell inspection techniques, have therefore suffered from a high 
false negative rate due to the tedium and faldgut imposed upon the technician. 

Several thousand women die each year in the United States alone from cCTvdcal 
cancer; a cancer from which a woman theoretically has a high probability of survival 
if detected in its early in situ stages. If not detected early, however, the chances of 
survival may decrease drastically. If a malignant cell in a pap smear is missed, by 
the time tiie woman has another pap smear performed the cancer may have advanced 
to its invasive stage from which a woman has a much smaller chance of survival. 
Consequentiy, the importance of detecting the presence of only one or a few 
malignant or piremalignant cells among the hundreds of thousands of cells in a smear 
cannot be ovCTStated. Unfortunately, present manual screening, methods are 
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inaccurate. In fact, rec^itly soitae laboratories have been found to have incorrectly 

■ ^ ' ■■ 

r"' " clasjsified as benign up to 30% of the specimens Containing malignant or premalignant 
ceils. Also unfortunate is that many prior attempts to automate the cell inispection or 
classification have been unsuccessfiil. 

Predominately/ these prior attempts at automation have relied on feature 
extractidh, template matching arid oth^ steitistical or algorithmic methods alone. 
These attempts have required expeaisive atid time-consuming cell preparations to 
distribute the cells and other dbjects over a 'Slide so that none of the cells or objects 
overlap. However, even then these attempts have been unsuccessful at accurately 
classifying specimens in a reasonable time fe^ 

These difficulties have been oivercome by combining an algorithmic or 
statistical primary classifier with a neural network based secondary classifier as 
disclosed in U.S. Patent No. 4,965,725, and U.S. Patent Application Nos. 
07/420, 105, 07/425,665, 07/520,611 and 07/6lt),423, which are incorporated in their 
ehtireties by this referdice. A commercially available automated pap smear screener, 
using a primary classifier in conjunction with a neurocomputer based secondary 
classifier is produced by Neuromedical Systems, Inc.* of Suffem, New York under 
trademark PAPNEr~^ 

SUMMARY OF THE INVEimON 
The present invention provides a method and apparatus for automating a cell 
classification process using at least primary = and secondiary classifications. 
Specifically, the primaify dlassification istep includes performing morphological 
"filtering on an image of a biologic^ specimen to detect cells having certain 
morphological features, sudi as a generally round shape or a certain size. The 

SUBSTITUTE SHEET 
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secondary classification step , then fiirdier classifies these objects using an 
implfementition of a neiml network. The present invention preferably performs at 
least two scans of the specimen to produce one image suitable for analysis by the 
primary and secondary classifiers and a secqnd image suitable for display on a high 
resolution color monitor to facilitate inspection by a cytotechnician. 

In accordance with thg present invention, a method of classifying cells based 
upon their morphology includes a nipthod . of classifying objects ^i^ a cytological 
spedmeil, including the steps of obtaining, a fir?t, inmge of at least part of a 
Cytological specimen, classifying objects in thC: first image on the basis of a 
predetermined criteria, obtaining a second image of at least one of the objects most 
likely to have a predetermined criteria, and displaying at least part of the second 
image to produce a visual display of at least one of the objects most likely to have a 
predetermmed criteria. 

In: accordance with another aspect of the invention, a method of classifying 
- objects ill a specimen includes the stq)s of obtaiiiing a first digital representation of 
at least part of the cytological specimen, storing the first digital representation, 
performing a first filteringoperation to filter out images in the repiresentation that are 
the approximate size of a malignant or premalignant cell or smaUer to produce a 
second digital representation, removing the images in the second representation from 
iht images in the first representation to produce a third rq>resentation, performing a 
' second filtering operation to filter out. images in the, first represenation that are 
smaller than the aii^^pximaite s^ of a premalignant or^malignant c^ll to produce a 
r . fourth representation, and elijaiinatir^^ tiie images in the fourth rq)resentation from the 
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images in the third lepresentationi to pibduce a lepiesentation having substantially 
only images the approximate size of a piemalignant or m^gnant cell. 

In accordance with a furth« aspect of the invention an apparatus for 
classifying objects in a qhtological spedm^ includes a device for obtaining a first 
image of at least part of the cytological spediten, a processor for classifying objects 
in the first image on the basis of a predetennined ciiteiia, a device for obtaining a 
second image of at least brie Of the objects most likely to have a predetennined 
criteria, and a monitor for displayihg at least part of the second image to produce a 
visual display bf^ at leist otfe Of the objects most lifcely to have the predetermined 
criteria. 

lihese and other objects, advantages, features and aspects of the present 
invention will become aHKoent as tiie following description proceeds. 

To tile accomplishmaits of tiie foregoing and related ends, the invaition, then 
comprises tii^ features hereinafter fully described in the specification and particularly 
pointed out in claims, the following deswiptibn and the annexed drawings setting 
forth in detail a certain illustrative embodimait of the invention, this bdng indicative, 
however, -bf-but o ne o f t h e various waysin-which-the principals of the invention may 
be employed. It will be appreciated tiiat the scope of the invention is to be 
determined by the claims and the equivalents thereof . ' 

BRIEF DESCRIPTION OF THE DRAWINGS 

In the annexed drawings: 

Figure 1 is a iSchematic illustritidn of a cytological classification or screening 
device in accoriiance with the present inveritioh; ■ 
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Figure 2 is a diagrammatic illustration of the scanning passes which the 
screeoing device performs; , 

Figure 3 is a schematic illustration of the screening device of Figure 1 with 
particular emphasis on the processinjg systein; 

Figure 4 is an illustration of the various image components representing areas 
^ of the specimen; 

Figures 5a through 5c are flowcharts illus;trating the prim^ classification 
function of the present invention;, and 

Figures 6a through 6d are graphical represen^tiqns of a morphological closing 
operation performed during the primary classification function. 

DETAILED DESCRIPnON OF THE INVENTION 

With reference to the several figures in which like reference numerals dq)ict 
like items, and initially to Figure .! , there is shown an automated cell classification 
device 10 in accordance with ih^ present invention. Briefly, the device 10 includes 
an automated optical microscope 12 having a motorized stage 14 for the movement 
. ofva slide 16 relative to the viewing region of the viewing portion 18 of the 
nncroscope, acameia 20iprt)btajning eic^^ from the optical microscope, 

a processing system 22 for classifying objects in the images as likely to be of a 
predetermined cell type^ and a meniory 24 and a high resolution color monitor 26 for 
the storage and display respectively of objects identified by the processing system as 
being likely to be of that predetermined cell type. 
. ; . In its preferred embodjnient the qlasisiflcation device 10 is completely, or 
nearly completely, autoin^ted.^ As such the microscope 12 will preferably include, 
in addition to the motorized stage 14, automated apparatus for focusing, for changing 
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lens objectives bttween high and low pow«, and for adjustment of the light incident 
of the slide, as well as circuitry for'contibllmg the movemait of the motorized stage, 
locally in lespcmse to a command frdm the processing system. The microscope 
may also include an automat^ slide tian^rt system for moving tfie slides containing 
the spedmai to be classified on to and bff of the motorized stage, a cell dotter for 
marking relevant areas of the slide, and a bar code reader for reading encoded 
information ironi the slide. Ah example of an automated microscope poforming at 
least some of these factions is manufactured by McBain Instraments, of Ca^ 

In accordance with the invention the automated mittoscope 12 preferably 
performs three scans of the slidfc having the specimen disposed theaeon, as shown 
diagrammatically in Figure 2. The first scan of the slide is paformed at a rdativdly 
low magnification, for «anq)le 50 power, and is called the low resolution scan (30). 
The second scan is performed at a higher magnification, for example 200 power, and 
is called the high resolution scan (35). The third scan is referred to as the high 
resolution rescan and is also performed at a high magnification (40) » 

During the first scan (30) of the slide, approximate focal planes for the 
specific areas-of the ^de aie found ^itis determined ^\1iether that area of the slide 
contains a portion of the specimen. Once a low resolution scan (30) has been 
performed of the whole slide, and the focal planes and areas of the slide containing 
the specimen have been logged, the high resolutidn scan (35) is performed. 

The Wgh rtisbliitioh a:ah 05) is peitfbrm 
in the low resolution ^can (30) to contain a portion of the specimen. Consequently, 
the comparatively long high resblutiori scan (35) is performed only on rdevant areas 
of tiie slide and the processing time is greatly reduced. During the high resolution 
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scan (35), the automated microscope 12 scans the relevant areas of the slide, and the 
' camera 20 takes- el/ectronic images of th^ areas and sends the images to the 
processing system 22. The processing system performs a primary classification of 
the image which finds the centroids of biological objects having attributes typical of 
the cell class for which screening js being performed, such as malignant cells. Using 
a smaller sub-image centered around these centroids, the processing system 22 
performs a secondary classification which assigns each centroid a value indicative of 
the possibility that the object having that centroid is a c?ell of the type for which 
classification is being performed^ Simultaneously, the ce^^ 
on the value assigned through the secondary classification. 

Upon completion of the Mgh resolution sc^ 
(40) is performed for the highest 64 ranked objects. During the rescan (40) the 
automated microscope 12 will move to each of the highest 64 ranked centroids and 
the camera 20 will obtain a high resolution color iniage of the object having that 
centroid. These 64 high resolution images, cdled color tiles, are then stored in the 
memt)iy 24 which may be a removable device, such as an optical disk or a tape, etc. , 
or a fixed st orag e d cvice^such^ a hard disk. "Alternatively, the sixty-four color tiles 
may be transferred to another computer via a network or through transportation of the 
data on a removable storage media. 

The sixty-four color tiles make \xp a summary screen which is preferably an 
8x8 matrix of high resolution, color tiles featuring a suspect cell in the center of each 
tile. It will be appreciate,; however, that other numbers of color tiles may be 
, displayed conciKrently to produce a summary screen, such as a 4 x 4 matrix. These 
sunmiary screens are displayed on the high resolution color monitor 26 for tertiary 
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analysis and classification by a cytotechnician. This analysis may take place at 
anytime after the highest sixly-four have been secohdaiily classified and ranked. 
Further, through the use of a removable memory device or a network connection, the 
images and tilM of the sunuiiay sciieert may be trawferr^ 
from the microscope 18, carriers 20 and pifocessihg system 22 for display and 
analysis In such an instance a separate gr!5)hics processor 41 (Figure 3) may be 
employed to drive the high resolution color monitor 26 and provide a suitable 
inta^ace with the cytotecluiician. 

A cytoteclmitiaii or cytotechnoldgist (hereinafter cytotechnician) can easily 
scan the summary sdreeh in seardh of an object having the attributes of the cell type 
for which classification is being performed. If the system is being used" to screen a 
p^ smear for the presence of cervical cancer, the cytotechnician would typically look 
for cells having attributes of malignant <rf premalignant cervical ceils, such as a 
comparatively large, dark nucleus. The cytotechnician may also make a 
determination from the summary screen as to whether a si)ecimen having no 
malignant or prennalligriant cells was taken properly. This can be done by ascertaining 
the pres e nce trf •endoceiTicai-t:disr-fiidocervical-cdb make up the lining of tiie 
transitional zone of die cervix where most cervical cancers start; consequently, their 
presence in a pap smear specimen indicates that the test swab must have made contact 
with the critical transitional zdne of the dervix. Since endocwvical cells have more 
characteristics common to a malignant cell than a vaginal or other benign cell has, 
in tiie absence of any true premaligriant dr'^haligfiant cells it will be ranked above tiie 
other cells and displayed. 
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Herein the screening, method and apparatus of the present invention will be 
described as used in screening a pap smear for the presence of cervical cancer cells. 
However, it will be apparent to a person of ordinary sldll in the art that this is only 
an illustrative use and that the present invention may be used in screening samples of 
other biological niatt^ taken by ^a variety of cell sampling techniques, such as 
aspiration and exfoliation to name but t;wo. Further it will be apparent that while the 
illustrative example screens for malignant or premalignant cells, the screening may 
be performed for the detection of other cell classes or types. 

Turning now to a more in-depth discussion of the preset invention with 
spedfic reference to Figure 3, the. screening device 10 is shown with particular 
empha^s on the classification elem^ts embodied in the processing system 22. The 
processing system 22 preferably, includes an image processor and digitizer 42, a 
neurocomputer 44^ and a gen^ processor 46 with, ppriph»als for printing, storage, 
etc. , 

/ llie general processor 46 is preferably an Intel* 80386 microprocessor or 

faster microprocessor based microcomputer although it ri^ 

device suitable forefficient exe cutioii of the-functions'described herein. Die general 

processor 46 controls the functionitig of and the flow of data b^een components of 

the device 10, nmy qaus^, ^ecution of additional primary feature retraction 
: algorithms and handles the storage of image and classification information. Hie 

general processor: 46 additionaUy controls peripheral devices such as a printer 48, a 
, storage device 24 such as an optical or magnetic hard disk, a tape driye, etc., as well 

as otfier devices such as a bar code reader 50, a slide marker 52, autofocus circuitry, 

a robotic slide handler, and the stage 14. 
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The image processor and digitizer 42 perfonns the pi^ 
functions described more fiiUy bdbw. In the preferred embodiment, the image 
processor and digitizer 42 is a commerdaUy available to^ 

extraction image classifier such as the ASPE5C Incorporated PIPE* image processor 
which includes among other thiiigs an image digitization function and an KMAP 
(Iconic to Symbolic Mapping) board. The PIPE» image processor is described more 
fuUy in U.S. Patent No. 4,601,055, the entire disclosure of wMch^ 
this reference. Alternatively, the image processing and digitization functions could 
be separated into twd'br more c»m^^^ Bdow,tiieimage piocessor and digitizer 
will be conjunctively referred to as tiie image processor 42. 

Secondary cdl classification is performed by die neurocomputer 44. The 
neurocomputer 44 is a computer embodim^t of a neural network trained to identify 
suspect cells. In tiiis embodiment tiie parallel structure of a two or three-layer 
baclq)ropagation neural network is emulated witii pipelined serial processing 
techniques executed on one of a host of commerdally avalable neurocomputer 
accelerator boards. The operation of tiiese neurocomputers is discussed in Hecht- 
Nielsen, -Robert, -^NeurocompntinF-^ckmg-tiie iiu^ IEEE Spectrum, 

March, 1988, pp. 36-41. The neural network is preferably implemented on an Anza 
Plus™ processor, which is a commerciafly available neurocomputer of Hecht-Nielsen 
Neurocomputers. Siich a neurocomputer could be easily configured to operate in a 
manner suitable to perform the secondary classification functions by one of ordinary 
skill in die art tfuroiigh reference tb coniesjwnding etc. Alternatively, 

secondary ceU classification fimctions' Could be performed using a template matching 
algoritiim designed to identify shapes knowh to be typical of a pathological ceH. A 
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template matching or otjier group processing algorithm could be efficiently 
implemented in a parallel distrifc|uted processing network, for example. Another 
alternative secondary classification embodiment is a holographic image processor 
designed fo perform grpup based classification. 

The image processor 42, the neurocpmputer,44, and the general computer 46 
may each access read-only, and/or random aqcess memory, as would be readily 
' apparent to one skilled in the art, for the storage and execution of software necessary 
to perform the functions, described relative to that processm^ component. Further, 
each component 42, 44, 46 includes (±-cuitiy,. chips, etc. for the control of 
communication or data transfer over the data bus 54 as well as other functions typical 
bf similar processors as WQuld be appreciated. 

Returning to a discussion of the operation of the device 10 and with reference 
to Figure 4, the area of the slide 16 possibly containing the biological matter of the 
specimen is segmented into a plurality of lov/s and columns, for example, 20 rows 
and 50 columns of equal sized blocl^s 60. Each block 60 occupies an area of the 
slide, for example, approximately ?000 microns x 1600 microns, and corresponds to 
an ittdividual image to bcAdcwcdme-by;one bylte^ Each block 60 is 

'Subdivided, for example, into sixteen eqiially sized analysis fielcjs 62. Each field 62 
is thus apprpxinmtely 500 nucrons by, 400 ,micrpns in size. Once digitized by the 
image processor. 4i2, each analysis field 62 will be represented by a 256 by 242 matrix 
or airay of pixels which corresponds to a. resolution of approximately two microns 
per pixel during a low resplution scan (30Xor high resolution scan (35), or a 512 by 
484 array of pixels corresponding to a one micron j^t pixel resolution during a high 
resolution rescan pass (40). Ez^ch pixel then represents the brightness, or gray scale 



wo 92/13308 PCT/US92/00^60 

-12- 

density of a discrete aiea of the analysb field 62 image. The gray scale density of 
each pixel is further represented by an 8 bit digital value; Consequently, each pixel 
will represent an ai^ of the analysis field image 62 by a gray scale level ranging 
from zero to 255. In opraation, the screening device will perform a lo\y resolution 
scan (30) on each analysis field 62 to determine if that field contains biological 
matter, and a high resolution sCan (35) on each of the analysis; fields 62 having 
biological matter fo detect objects contained therein which are likely to be malignant 
or premalignant cells.- A third scan (40), the high resolution rescan, may also be 
performed on an analysis field 62. or a portion of an analysis field, if during the high 
resolution scan (35) the procesdng system found an object within the field which is 
litely to be a malignaint or premalignant cell. 

During the low resolution scan (30) the objective of the microscope 12 is set, 
for example, at its 50 magnification power, and tiie microscope begins scanning tiie 
individual blocks 60 of the sUde 16. For each block 60 the microscope 12 will 
autbmaticaUy determine the jqjproximate focal plane for tiiat area of th^ slide 16. As 
the cover slip covering the ^pedmea tends to be somewhat wavy or possibly angled, 
such as due^ ail-bubbles -ttntamed mi Jei Uie t u v erdiprthe fogal plane may vary 
from block 60 to block. Once the focus is determined for the block 60 being viewed, 
the camera 20 will capture the image of tiie block and send tiiat image to ttie image 
processor 42 tiiroiigh a suitable digitizer. The image processor 42 tfien subdivides 
tiie block 60 into analysis field 62 and determines whettier tiiere are areas of interest 
in each analysis field corresponding to objects which may be biological material. If 
a field 62 contains material which may be biological, tiie block 60 is identified along 
witii its approximate focal plaie.and stored in memory for future analysis during tiie 
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high resolution scan (35). This low resolution scan (30) should be performed for all 
blocks 60 on tiie slide 16. 

Once the low resolution scan (30) has. been completed and all of the blocks 60 
containing objects which are possibly biological material have been identified in 
memory, the high resolution scan (35) is begun. Initially, a scan path is determined 
which will allow the microscope 12 to view each block 60 possibly containing 
biologicilmatterpreferably with theleMt anaountof mo For 
the high resolution scan (35), the objective cprre^nding, for example, to a 200 
pbwer niagnification is inserted into the viewing i»tii of the miCToscope 12, and the 
scan is begun at the first block 60 in the scan path. The microscope 12, via the 
motoii2ed stage 14 will move tiie slide, 16 into a position such that the first block 60, 
which was identified as haying biological matraal during the low resolution scan, wiU 
be in the field of view of the microsa^. The nucroscope 12 will thai, based 
initially on the focal plane determined during the low resolution scan (30), focus the 
t)l(>ck 60 under the high resolution magni$cation level. The block 60 is digitized and 
again subdivided into 16 analysis fields 62. Hie image processor 42 will tiien 
p^orm the primary classification of the objects in eadi analysis field 62 as discussed 
TOOK fully l)elow. This primary classification finds the centroids of objects in each 
field that have the correct size and gray scale density characteristics. 

When an object in an analysis fid^l 62 has been identified as having the size 
' aiid gray scale density characteristics of p premalignant or malignant cell, a 24 x 24 
array of pixels surrounding. the object centroid, called a net image 64, is transferred 
to the secondary classifier for fiirther classification, A net image 64 is approximately 
48 X 48 microns in size sit a resolution of 2 microns per pixel. As a malignant or 
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premalignant cdl nucleus tends to lange between 10 and 40 microns in diameter, the 
net image 64 is sufifidenUy large to contain a complete image of a suspect cell. 

The highest 64 ranked objects arc displayed on the summary screen 66. As 
discussed above, the summary screen may bie an 8x8 matrix of 64 discrete images, 
called color tiles 68, a 4 x 4 arrangement of 16 color tiles, or some other 
arrangement. Color tiles 68 arc bbtained during the rescan pass (40) . Each color tile 
represents an approximately 128 x 104 micron area surrounding the centroid of a 
suspect ceE, vwth a iBS»toti6n of one mi^^^ Each color tile produces a 

high resolution color iiiiage of a suspect ceU and surrounding ceUs and biological 
matter, with the suspect ceU caitered in the tile. By reviewing the summary screen 
66, the cytotechiiician can relatively easily classify the high resolution color images 

of the suspect cells. 

It wiU be appreciated by a person of ordinary skUl in the art that while 
particular resolutions and image sizes were described above, these particular values 
are exemplkry. It will further be appreciated that different resolutions, image 
parameters, magnification levels, etc., can be employed to accompHsh the same or 
similar-fesiilts as titepafticttlarembodraient of iheinvention described above, and that 
all such differing resolutions, image parameters, etc. are within the scope of the 
present invention. 

Turning to a more ih-depth discu^sioh of the primary classification routine, 
and referring to Figures 5a-5c, there is shown a flowchart of the primary 
classification functions which the image prbcestor 42 performs. Numbers contained 
within parenthesis below Mirespond to iiie numbered steps in the flowchart. 
Similarly, letters contained within parenthesis (denote the flow of image data at 

eiiRCTiTllTE! SHEET 
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various Steps in the flowchart. Once a block image 60 has been focused and taken 
by the camera 20, the image processor 42 will digitize an area of the block 60 
corresponding, to an analysis field 62 and grab the 8 bit gray scale red and green 
images from the camera (100). The red and green 8 bit gray scale images are then 
combined to produce a monochrpme image (1Q5) which is stored in a segment of 
memory of the image processor 42, called the frajne buffer. The Papanicolaou stain 
used in tx^ating a . pap smear dyes the nuclei of biological cells within the smear a 
purple color. Since the colors red and green y^rhen com^^ 
- make a yellowish green color which is du-ectiy opposite the color purple on a color 
space triangle, combining red and green, without blue, creates an iniage wherein the 
purple stained nuclei appear very dark and the areas of other colors appear brighter. 

Briefly, the primary classifier performs a morphological "well" algorithm 
which filters out objec^ that are the size of a premalignant or malignant cell or 
smaller. (A "w^" algorithm is the inverse of a morphological "top hat" algorithm.) 
The resulting image, containing only objects which are too large to be a cancerous 
< cell, is then subtracted from the original image containing all of the objects, 
CpMcqtiCRfly,-Witot- i5 lefl -are-t)b;ect$ of ;th^ correct size or smaDer. A separate 
imag|5 is then prepared from the original image which will contain only objects which 
are too small to be of concern. When this latter image, containing only objects which 
^ are>too ,small, is tbep subtracted from the ima^e having objects of the size of a 
possible cancerous cdl or: smallejc, the resultant image vvill thus contain only images 
; . being of the size of a i^ssiblej^ancerojis cell.. The cenfroids of the objects in this 
T iniage^ are then determined! aiid Uie images centered around those centroids are sent 
to the sepondary clasafier for fiirttier classification. 
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The image in the frame buffar, which corresponds to an analysis field 62 and 
is lefened to heron as the frame irnage, is spatiaUy filtered such as with a gaussian 
low pass filter to remove random noise ftom the image (110). Preferably, the 

gaussian filter has a convolutioh ifitask of: 

■ • ' -.-^ 12 1 

2 4 2 

■■■■■■■ ■ ■■■ 1 21 ^ 

This convolutictti mask is tnb\^6d across To explain, 

the convoluSon iriai^ will W initially centered on the first pixel in the frame image 
pixel matrix. Consequently, the 8 bit gray value of this pixel will be multipUed by 
4 while the 8 bit gray values of the pixels immediately above and bdow and on either 
side will be multiplied by 2, and the adjacent diagonal pixels.will have thdr 8 bit 
gray scale values multiplied by 1. All nine of these results are summed and the result 
is divided by 16 and placed in a pbcel location in a result frame corresponding to the 
location of the center pixel. The convolution mask is then shifted to the next pixel 
in the frame image pixel matrix and the operation is repeated. This continues for all 
pixels iiHhe-matrix7-thus-gcnerating-a-conqjiete resultiiame. 

A morphological closing is then performed on the frame image in the result 
frame to filter out all objects smaller than a certain diameter which is somewhat 
larger than the size of a malignant cell nucleus. Consequently, what is left are only 
those objects which are too large to Correspond to malignant cell nuclei. This 
morphological closing is performed using a series of gray scale dilation operations 
followed by a series of gray scale erosion operations. Gray scale dilation is a 
mathematical morphology term used in image processing to denote an operation 
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; wherein a mask is centered on a pixel and a corresponding pixel in a corresponding 
result frame pixel is replaced by the largest value of its neighboring pixels added to 
their corresponding mask values or itself added to its conesponding mask value. 
Erosion is a similar term wherein ^e center pixel is replaced by the minimum value 
of its neighbors added to their corresponding mask values or itself with its 
corresponding mask value added to it. 

In the prefixed embodiment of the invention, gray scale dilation operations 
are performed using an 8 neighboring pixel mask wherein the center pixel and its 8 
ndghboring pixels are each allocated equal weights, followed by a gray scale dilation 
using a 4 neighboring pixel mask wherein the pixels diagonally related to the crater 
pixel are ignored and the remaimng 4 neighboring pixels are allocated equal weights 
with the center pixeL The respective masks for these dilation operations would 
appear as: 
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For the CTOsion operations, a gray scale erosion operation is first performed using a 
: 4 neighboring pixel niask followed by an erosion using an 8 neighboring pixel mask 
where all the neighboring pixels are taken into account. The respective masks for the 
erosion operation would appe^ as: 
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The combinatioii of operations using these masks moiphologicaUy approximates an 
octagon, which is digitally analogous to the morphology of a nucleus which tends to 

be goierally round. ' 

The manntt'im which this morphological closing fflt^ 
image is graphically represented in Figures 6a-d. Figure 6a illustrates two objects; 
the object on the left 70 is a large dark nucleus and the object on the right 72 is a • 
smaller, less dark cell, such as a leukocyte. The horizontal line 74 rq»resents a row 
of pixels passing through the objects 70, 72. The gray scale values for this row of 
pixels is shown in Figure 6b. The large dark nudeus 70 forms a wide and deep gray 
scale rectangle 76 due to its relatively large size arid darkness. The leukocyte 70, 
being smaller and less dark, forms a narrower, shallower rectangle 78. 

As dilation Operations are performed, these gray scale rectangles are gradually 
fiHed in arindicated in Figure 6c. After one such dilation operation, the large dark 
nucleus may now be represented by the relatively wide depression 80 while the 
leukocyte' may be rq)resehted by the narrow spike 82. Subsequent dilation operations 
will continue to fill in these depressions. Consequently, a relatively small object will 
be completely filled in while larger, darker objects, such as malignant nuclei, will 
renwin as OTmewhat nanx)wer depressions. ■ ' 

When the image is then subjected to a series of erosion operations, the narrow 
depressions which are the gray scale representations of large, dark objects will be 
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expanded to approximately their original size and shape, whereas the completely filled 
in gray scale lepresoitations of small objects will not be eroded. The morphological 
closing will thus yield a row of pixels having gray scales, as shown by the solid line 
84 in Figure 6d, which represent the large dark object, but have the object which is 
too small to be a malignant or preroalignant cell fUteied out. 

A discussion of mathematical morphology pan be found in Serra, "Introduction 
to Mathematical Morphology" Computer Vision, Graphics, and Iniage Processing, 35, 
283-305 (1986) and Sternberg, "Grayscale Morphdqgyr , Computer Vision, Graphics, 
and Image Processing, 35, 333-355 (1986). 

Returning to tiie flowchart illustrated in Figures 5a-5c, a gray scale dUadon 
is^jaformed on the gaussian Ipw pass filtered frame iniage using an 8 neighboring 
pixel mask (1 15). , A gray scale dilation is then performed on this dilated frame image 
using a 4 neighboring pixel mask (120), Ttiis series of gray scale dilations is then 
performed one Additional, time (125, 130). 

The dilated image is then eroded using a ,4 neighboring pixel mask erosion 
foUbwed by an 8 neighboring p^el niask erosion (160). Again, this series of ax>sions 
is'pCTformed'Oi ie additi o nal t ini e-(165,- 17^^ . ' 

= It will be appreciated by a person skilled in the art that other filtering 
fuhctions could be performed to adhieve similar results, such as a rolling ball or cone 

'■ filterr " - . •.- ; 

: ; • . : The resulting dilated-eroded frame image has npw had the smaller objects 
filtered out and consists only of objects which are larger than a malignant cell 
■ nucleus. TMs. dilated eroded .foune image is then subtracted from the firame image 
(B) which has been gaussian Altered but dilated and ^oded (195). The resultant 
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frame image thus consists only of objects which gmeially correspond to the size of 
a malignant or piemalignant nucleus or smalli^. 

A threshold o^mtion is Aen performed on. this image with pixels having a 
gray scale value above a certain threshold bdng assigned a binary ' 1 ' and those pixels 
have gray scale values below that thre^old being a^gned a binary zero (200). The 
threshold is chosdi io isiter out objects which are not dark iaiough to be nuclei. A 
binary 3x3 erosion is Aeii pMformed on die iriiage to remove the out« pixel 
boundary of die remilinflig objects, thus making the objects narrower by one pixel in 
all directions (2CK5). * This binary erosion is accomplished by performing a boolean 
AND on the center pixel and the ei^ht neighboring pixels. Rq)resenting the mask as 
below: 

A B C ■ 
D E F 

■ ■ G H I 

Mathematically, this will be rq)resented as A & B & C & P & E& F & G & H & 
I, where each letter indicates a binary value at certain pixel locations. As discussed 
rfxjve relative'to the ga nssian-iow paiis filtei , xtilatiDmnd'efosion'^operations; this 
mask slides across the frame so that the binary aosion is pwformed with the mask 
evaitually centered oil each pixel in the frame, thus creating an eroded result frame 
linage. 

The resulting image is theri boolean ANDed (210) with the original frame 
image (C) which has been tfiresholded (215) as described above to produce a binary 
image. Boolean ANDing these'iiriageis masks off objects that are of the jrppropriate 
size but are too bright to be malignant or premalignant cervical cells. An example 
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of the objects which would be eliminated from the frame image by this operation are 
clumps of red blood cells. 

The frame image now contains objects of the appropriate gray scale density 
that are of a sizeMCpiresponding to. a maUgnant or preni^ cervical cell and 
smaller. These snialler objects typically represent neiutrophils which are white blood 
cells, Consequently, these objects which are smaller than a cancerous cervical cell 
nucleus must be removed fix)m the frame im^ge. Since the frame image is now in 
a binary format, a binsuy mask is nepded to niask off the smaller objects. 
Accordingly, the mask would have binary 'I's in all locations of the frame image 
v pixel array excqpt for those locations, occupied by a small image, where there would 
be zeros. Consequently, by boolean ANDing the binary frame image with the binary 
mask the small objects are removed and predominantly what is left are objects of the 
appropriate gray scale density and size for a cervical cancer cell nucleus (220). 

This mask is obtained by taking the untreated frame image (C) stored earlier 
and treating the image with a well-known Spbel operator to find areas within the 
imag6 (225) having relatively large arithmetic gradients. Areas of high detail in the 
imager^ aieas4taving-retetivelyrsniail objects,- yili-irave large gradients.' The Sobel 
operator, which is essentially a high-pass image filter, will pass those areas having 
large gradients and reject the areas of large objects which have smaller gradients. By 
choosing the correct filter parameters, the edges of objects are found. The image is 
i: then thresholded tp conyeft the gray scale image into a binary image (230). A 3 x 
3 binary dilation is then perfomjied on the image to slightly expand or thicken the 
^ edges in :the im2ge.(235). Jhe.binary dilation operation is performed as a boolean 
'OR' operation. Using the niask potation discussed earlier relative to binary erosion, 
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mathematically the binary dilation would be esKpiessed as A|B|C|D|EjF|G}H|I. 
The dilation operation is then repeated preferably lising only the adjacent pixels. 
Mathematically tlus second dilation would be represented as B|D|E|F|H (240). 
After the edge thickening operation/objects small» 

become solid *blbbs" and objects larger than this diameter will remain open in the 

center. This function c»irid also be perfomed with a gray^^ 

or top hat filter as described aboVe with its parametax set to find smaller objects. 

A number of the outermost rows and columns of pixels; for example, dght, 
forming a border around the frame are^ then removed, as thdy contain artifacts and 
other iirelevkit infonnation introduced by the operations performed above G45) . The • 
complement of theresultant image is then takeii by a boolean 'NOT' operation (250). ^ 
Consequently, the binary image will consist of binary 'I's in all locations exc^ in 
the slightiyailarged areas encompassing objects too smaU to be malignant cells. This 

complemented image thuis forms the binary mask used to subtract the small objects 
from the earlier developed frame im^e. When boolean ANDed with the frame 
image having objects of the 25>propriate size and smaller (220), the small objects will 

bediminated. " ' ' 

Once the small objects have been removed and the frame image contains 
predominantly objtets having a gray scale drasity and size compatible with the gray 
scale density and size of a cervical cancer cell nucleus an operation is performed to 
suppress isolated white pixels in the frame image (255). These isolated white pixels 
(binary '1') constitutes noise introduced dnting any of the previous operations and are 
' not of a sufficient size to be a cell nucleus. Using the binary inask notation discussed 
above relative to binary erosion and diiitibn operations, the suppression operation can 
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be expressed mathemticaUy as (E & A)|(E & B)|(E & 
: G)|(E&H)|(E&I). 

: : Once the isolated pkels Imve^b^ suppressed, a "shrinking" operation is 
performed four^times (260, 265, 270, 275). "Shrinking" is a boolean operation well- 
known in the field of image processing. The shrinking operation successively 
removes layers of pixels a^und the object, in the image frame until, for an object of 
the appropriate si?e, only one pixel or no pixels, remain. In the case where the object 
is completely removed, the last pixel removed is, replaced ydih a binary L Since the 
outer layers of pixels of the objects were successively removed progressing inwardly, 
the remaining or replaced jnxel will represent the approximate center or centroid of 
^ the object. Any objects remainiiig in the image which are larger than one pixel are 
a removed, as they correspond to something which is larger than a cervical canc^ cdl 
nucleus would be (280)* 

The location of the remaining single pixels, as defined by their X and Y 
coordinates, on the screen, are then listed and recorded in memory (285). The 
ISMAP board of the PIPE* iniage processor conveniently provides these coordinates. 
The identification-of t h e i » ili oids u f ubjects^wlitelrec^ to the same size and 
gray sc^e density that a typical cervical canc^ ceil would be expected to have marks 
theend.of the primary classification phase of Qpeiatip^ 

. Given the di3Cussion above and a general knowledge of image processors, such 
, as the preferred PIP^*' image processor, tiie primary classification functions could be 
reduced to the apprppr^te. so)^ 

, As would be apparent, tlje codp ^puld i)e loaded, bumed, or otherwise encoded into 
memory apc^&ssible by the image processor 42 fqr execution by the image processor. 
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A 24 X 24 ariay of pixels sunbunding each centroid, identified earlier as a net 
image 64, identified by the primary classification fimction of the image processor 42 
is then transfened to the general processor 46 for storage. A. secondaiy classification 
of the objects lepresoited by these net iioaages 64 may then be performed. 

The geh^ laocessor 46 individually transfers each net image 64 to the 
neurocomputer 44 for secondary clajsification. The task of the secondary 
classification is to distingiush the premalignant and maHgnant cells from other objects 
of the same size w^ch niay pass the primary classifier, such as.ceU clumps, debris, 
clumps of leukocyte aid inucus. N 

Basfed on training petformed with a training set of several hundred or . 
thousands of known benign and premaUgnant or maUgnant ceUs, as described more 
fiilly above, the neurocomputer 44 wiU assign each net image 64 with a value, called 
a net value, ranging from .1 to .9, as determined by the likdihood that the object is 
a premaUgnant 6i maUgnarit ceU. One major advantage of the present invention over 
prior known ceU classifiers resides in the fact that each net image 64 presented to the 
secondary classifieif is pra»ntered through the primary classification on the coitroid 
of the suspect ceH nucteus, as^escrifaed^bove. In pieviuui. kuuwu -attempts^to utflize 
neural networks and other high level template matching pattem classifiers for image 
recognition, difficulty has been encountered in consistently presenting the classifier 
with the centroid of the image requiring classification. To use an example firom 
another application doriiain, back propagatibn networks; are excellent at reading 
handwrittai zip icode digits but have difficulty in finding where the zip code is on the 
envelope. The present inventiion overcbnies this difficiilty in the domain of cytology. 

Another advantage of the present invention is that during actual classification 
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operations, a secondary classifier is presented with precisely the same type of net 
images on which it was trained. These images are also centered on the centroid of 
the suspect nucleus by the primary .classifier in a maimer identical to that used to 
prepate the training set images. This mates a generalization task of the secondary 
classifier far easier and niuch more isjiccessful than anything previously known. 

It should be recognized; that while the image processor and digitizer 42, the 
general processor 46, and the neurpcompyter 44 are d^ribed operating in a serial 
manner, in^ actual practice as many ^functions will be performed in paraUel as is 
possible. Consequently, the components 42, 44, 46 may process different slide 
segments or different areas of a segment concurrently^ greatly reducing the time 
required to screen a slide, i : 

As noted above the secondary classifier is trained to associate a known benign 
image with an output. of .1 and a known pathological image with an output of .9. 
Such ou^uts represmt, for example, the degrese of certainty that a ceU is normal or 
abnormal, respectively. When the secondary classifier is presmted with new, 
' unknown cells, it generalizes from its training and attaches a net value to the image. 
The closer thai theMccmdary dassifieris abte^ 

the benign category,, the doser is its.net yalue equal to .1. Conversely, the more 
closely that the unknown image , appears to re»mble the nonbenign images of its 
training set, the closer is the net value assigned to that image equal to .9. 

Once all objects classified by the primary classifier to be possibly premalignant 
' ' or malignant have be^ cJlassified by thp secondary classifier, the net values assigned 
to those objepts by the5eco|)dary classifier are ranked firom closest to .9 down to .1. 

' ' ■ ' ' . ' * I "i ■ " ; " - 
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The highest ranked 64 objects are then stored. This completes the end of the high 
resolution scan (35). 

Once the highest ranked 64 objects in the specimen have been identified which 
are most likely ta be malignant or pranalignant, the Mgh resolution rescan (40) is 
begun at the 2(X) power magnification. During the rescan (40) the stage 14 will move 
the slide relative to the microscope 12 so that one of the highest 64 ranked object is 
in the viewing path of the mieroseppe. The image is then focused according to the 
previously determined high resolution focus parameters, and the camera grabs, at 512 
X 484 resolution, the 128 x 104 red, green and blue component image around the 
centroid location. This high feSdlution color tile 68 is then stored in the memory 24, 
such as on an optical disk or tape. These operations are then performed for the next 
cell until all 64 of the highest ranked cells have been rescanned and their high 
resolution color images have been stored in the memory 24. This completes the 
rescan pass (40). The automated classifier 10 may then remove the slide and replace 
it with another slide for fiirther clasaficatibn. 

Once stored, the 64 color tiles 68 may be displayed as a summary screen 66 
in their descending order of ranking, their positioned relation to each other in the 
specimen, or in some other arrangement. The tertiary classification performed by a 
cytotechnician can th6n take place at any time. ■ 

STATEMENT O F INPUSTRIATv APPLTCATION 

The present invention is applicable to cell classification in general and is 
particularly applicable to the classification of cdls in a pap smear. 

SUBSTITUTE SHEET 
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What is claimed is: 

1. A method of classifying objects in a cytological specimen, comprising 
the'stq)s of: . 

a) obtaining a first image of at least part of such cytological specimen ; 

b) classifying objects in such first image on the basis of a predetermined 
criteria; ; 

c) obtaimnga second image of at least one of such objects most likely to 
; have such a predetCTHiined criteria; and . 

d) ; displaying at least part of sudi second image to produce a visual 
. 'idisplayofatleast^^^ to have such predetermined criteria. 

2. The method jof claim 1, wherein such first image is of a lower 
resolution than sud) second inu^e. 

3. llie method of claim 1, including the step of fiirther classifying such 
objects in such visual' display. 

4. The method of claim 1, wherein such visual display represaits plural 
■; 'Objects.- ■ , ,, > 

.5. The method of claim 4, including the step of arranging such plural 

objects in such visual display in an eight by eight array- of sixty-four elements. 

6. The method of claim 4, including the step of arranging such plural 
objects in such visual display in a four by four array of sixteen elements. 

7. Thfe method of plajjii 5 , ^wherein a caitroid of each object displayed is 
approximately centered in one pt such elements, 

8. r The method of claim, 6, wherein a centroid of each object displayed is 
approximately catered in one of such elements. 
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9. ThemethodofclaimS, wherein such pluial objects are arranged in an 
order related to the likelihood thatieach object has such, predetermined criteria. 

10. The method of claim 5, whaein such pjural objects are arranged 
according to their idativefwsitiohalJocations in such^cytoto^ 

11. Aniethodofclassifj^gobjectsinaspesdmenip^^ 

^ obtaining a first digital representation of at. least part of such 

cytological spedmen;' 

b) storing such first digital. reiHeserttation; 

c) performing a first filtering operation to fillter out images in such first 
representation that are the approximate aze of a malignant or premaUgnant cell or 
smaller to produce a second digital' rq)resentatiQn; , 

d) removing the images in such second representation from the images in 
such first represMitation to produce a third n^resenlation; 

e) performing asecond filtering operation to filter out images in such first 
representation that are smaUer than the approximate size of a premalignant or 

malignant cell to produce a fourth representation; and 

f) eliminating the images in such fourth representation from the images 

in such third representation to produce a further representation having substantially 
only images the approximate size of a premalignant or malignant cell. 

12. The method of claim 11, wherein , said step of performing a first 
filtering operation includes performing a morphological well function. 

13. The method of claim 11, wherein said stq) of performing a first 
filtering operation includes performing a morphological tbp hat function. 
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14. The method of claim 11; wherein said step of performing a second 
filteiiiig operation includes performing an azithnietiQ gradient operation on such first 
digital representation. . ^ 

15. The method of claim M wherein said step of performing a second 
filtering operation includes performing a moiphological well function on such first 
digitad representation. 

16. The method of claim 11 wherein said step of performing a second 
filtaing operation includes perforndng a^morphologieal top hat function on such first 
dig;ital representation, ; 

17. The method of claim 11,; wherein said step of removing includes 
masking out images which ape too bright to be malignant or premalignant ceUs. 

18. The method of claim 11, wherein said step of eliminating includes 
taking the complement of said fourth representation and boolean ANDing the resulting 
represehtatibn >^dth such third representation. 

19. The method of claim 18, wherein said stq) of taking the complement 
includes performing a boolean NOT, 

20. The method of claim 13 , wherein said step of performing an arithmetic 
^adient operation includes performing a Sobel operation, 

21. An apparatus for classifying olyects in a cytplogical specimen, 
' fciomprising: means for obtaining a first image of at least part of such cytological 

specimen;' -ri- .w : . 

means ; for ;dassifying objects in such first image on the basis of a 
predetermined criteria; 
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means for obtaining a second image of at least one of such objects most 

likely to have such a predetermined ctiteiia; and 

means for displaying at least part of such second image to produce a 
visual display of at least one of such objects most likely to have such predetermined 
oiteria. 
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